Affect-Aware Tutors: Recognizing and Responding to Student Affect

Theories and technologies are needed to understand and integrate knowledge of student affect (e.g., frustration, motivation, and selfconfidence) into models of learning. One goal of this research is to integrate tools that model student affect into intelligent tutors and enable tutors to elicit, sense, communicate, measure and respond to student affect. This article presents multiple solutions towards this goal and discusses systems that begin to redress the cognitive vs. affective imbalance in teaching. We describe our broad approach, how we will evaluate the impact of affect intervention on student learning and three main development objectives: i) affect recognition, ii) interventions in response to student affect, and iii) emotionally animated agents. 1 Vision and challenge Intelligent tutors provide individualized teaching in multiple domains and demonstrate learning gains similar to or greater than those provided by human tutors (Fletcher, 1996; Koedinger et al., 1997; Shute and Psotka, 1995). However, much previous research has tended to privilege the cognitive over the affective in which theories of learning view thinking and learning as information processing, marginalizing or ignoring affect (Picard et al., 2004). If computers are to interact naturally with humans, they must recognize affect and express social competencies. However, the role of affect in instruction is at best in its infancy (Picard et al., 2004). One obvious next frontier in computational instruction is to systematically examine the relationship(s) between student affective state and learning outcomes (Shute, 2008). This research looks at the role of student affect in learning and the role that new technology plays in recognizing and responding to affect. We describe research to measure, model, study, and support the affective dimension of learning in ways that were not previously possible. Our systems are used in the classroom with the goal of improving learning and perseverance. Affective interventions encourage learning, lessen student humiliation and provide support and motivation that outweighs or distracts from the unpleasant aspects of failure. This research is based on efforts at the University of Massachusetts, Arizona State University and the MIT Media Lab. The next section describes theories of affect, learning and human emotion. We look at the constellation of student behaviors that we label as emotion and provide a brief overview of how computers can recognize and respond to student affect. The next three sections describe three approaches to affect recognition (human observation, hardware sensors and machine learning Submitted to special issue on Modeling and Scaffolding Affective Experiences To Impact Learning, International Journal of Learning Technology edited by Rana el Kaliouby and Scotty Craig. 2 techniques), several interventions that respond to a student’s cognitive-affective state and two sets of emotional embodied pedagogical agents. The final two sections provide a discussion and a view of future work. 1.1 Theories of affect, learning and human emotion Modeling student emotion has become increasingly important for computational teaching systems and emotion has been named as one of the twelve major challenges for cognitive science (Norman, 1981). Human emotion is often defined as an intuitive feeling derived from one’s circumstance, mood or relation with others. Teachers have long recognized the central role of emotion in learning and the extent to which emotional upsets can interfere with mental life. Student interest and active participation are important factors in the learning process (e.g. Bransford et al., 2000). Students learn less well if they are anxious, angry, or depressed; students who are caught in these states do not take in information efficiently or deal with it well (Burleson and Picard, 2004; Picard et al., 2004; Goleman, 1995). Teachers often devote as much time to the achievement of students’ motivational goals as to their cognitive and informational goals in one-to-one human tutoring situations (Lepper and Hodell, 1989). Emotions can paralyze a student’s ability to retain information about a task (Baddeley, 1986). Several studies have addressed emotions involved in learning (e.g. Lepper and Chabay, 1988; Mandler, 1984; Kort et al, 2001). Human emotion is completely intertwined with cognition in guiding rational behavior, including memory and decision-making. The human brain is a system in which emotion and cognitive functions are inextricably integrated (Cytowic, 1989). Emotional skills have been shown to be more influential than cognitive abilities for personal, career and scholastic success (Goleman, 1996). For instance, in the comparison of impulsivity and verbal IQ as predictors of future delinquent behavior, impulsivity was twice as powerful a predictor (Block, 1995). Recent findings suggest that when basic mechanisms of emotion are missing, intelligent functioning is hindered. Nearly a hundred definitions of emotion have been categorized as of 1981(Kleinginna and Kleinginna, 1981). Yet no comprehensive, validated, theory of emotion exists that addresses learning, explains which emotions are most important in learning, or identifies how emotion influences learning (Picard et al., 2004). Most studies of emotion do not include the phenomena observed in natural learning situations, such as interest, boredom, or surprise. Rather, emotion definitions emphasize cognitive and information processing aspects and encode them into machinebased rules used in learning interaction, e.g., OCC model of emotion (Ortony et al., 1988). Acceptance of ideas about emotion in learning is based largely on intuition and generalized references to constructivist theorists (Piaget and Inhelder 1969; Vygotsky, 1962, 1978). These theories discuss how to motivate, engage, and assist students in a general way. Yet, they do not provide descriptions at the level of individual human-to-human interactions and clearly do not provide methods suitable for implementation in intelligent tutors. Motivation is one emotion strongly linked to learning and has been defined as a person’s direction, intensity and persistence in an activity. Students with high intrinsic motivation often outperform students with low intrinsic motivation. A slight positive approach by a student is often accompanied by a tendency toward greater creativity and flexibility in problem solving, as well as more efficiency and thoroughness in decision-making (Isen, 2000). If student motivation is sustained throughout periods of disengagement, students might persevere to a greater extent through frustration (Burleson and Picard, 2004). Studies of motivation in learning consider the role of intrinsic versus extrinsic influences, selfefficacy, students’ beliefs about their efficacy, the influence of pleasurable past learning experiences, feelings of contributing to something that matters and the importance of having an Submitted to special issue on Modeling and Scaffolding Affective Experiences To Impact Learning, International Journal of Learning Technology edited by Rana el Kaliouby and Scotty Craig. 3 audience that cares, among other factors (Vroom, 1964; Keller, 1983; 1987; Ames, 1992; Vail, 1994; Bandura, 1977; Pajares, 1996; Schunk, 1989; Zimmerman, 2000). Theories of motivation are often built around affective and cognitive components of goal directed behavior (e.g. Dweck, 1986, 1999; Dweck and Leggett, 1988). Flow, or optimal experience is often defined as a feeling of being in control, concentrated and highly focused, enjoying an activity for its own sake, or a match between the challenge at hand and one's skills (Csikszentmihalyi, 1990). In direct contrast Stuck, or a state of non-optimal experience, is characterized by elements of negative affect and includes a feeling of being out of control, a lack of concentration, inability to maintain focused attention, mental fatigue and distress (Burleson and Picard, 2004). The phenomenon of “negative asymmetry” or the staying power of negative affect, which tends to outweigh the more transient experience of positive affect, is also an important component of learning and motivation (Giuseppe & Brass, 2003). The concept of affect is often distinguished from that of emotion. Affect refers to an observed emotional state or biological response to an external stimuli. Incorporating affect within humanhuman interactions is very powerful. In their research on “thin slices,” Ambady and Rosenthal demonstrated that when participants were shown a short segment of video, as little as six seconds of a teacher’s first interactions with a student, they could predict that teacher’s effectiveness and student end-of-term grades (Ambady and Rosenthal, 1992). Wentzel has shown that caring bonds between middle school children and their teachers are predictive of learners’ performance (1997). Some researchers have raised the concern that one cannot begin to measure or respond to emotion until a clear theory of emotion is articulated. However, even without a fully-fledged theory of emotion, computers can be given some ability to recognize and respond to affect (Picard et al., 2004). In fact, research shows that efforts to build models of a less understood phenomenon will aid in improving the understanding of that very phenomenon (Picard et al., 2004). Thus we simultaneously engage in both the practice and the theory directly related to developing affect-aware tutors in an attempt to advance both. Ekman’s Categorization Cognitive-Affective Term Emotion Scale High pleasure Joy Low pleasure “I am enjoying this.” . . . “This is not fun.” Frustration Anger Low-frustration “I am very frustrated.” . . “I am not frustrated at all.” Novelty Surprise Boredom “I am very hooked.” . . . “I am bored.” Anxiety Fear Confidence “I feel anxious” .. . . “I feel very confident” Table 1. Cognitive-affective terms based on human face studies (Ekman et al., 1972; Ekman 1999) 1.2 Categorization of emotion We identify a subset of emotions that we intend to recognize in student behavior and for which the tutor will provide interventions during learning. This selection of emotion is based on both cognitive and affective analyses. We begin with Paul Ekman's categorization of emotions based on analyses of facial expressions that includes joy, anger, surprise, fear, disgust/contempt, and surprise (Ekman et al., 1972; Ekman, 1999). However, we realize that these emotions are appropriate for general-purpose description and are not specific to learning. Emotions referred to by students and teachers in a learning environment tend take on a slightly different flavor. Submitted to special issue on Modeling and Scaffolding Affective Experiences To Impact Learning, International Journal of Learning Technology edited by Rana el Kaliouby and Scotty Craig. 4 To address this, we added a cognitive component to Ekman’s categorization that is present in educational settings, thus initiating what we call “cognitive-affective” terms. For each of Ekman’s emotion we created a scale, resulting in four orthogonal bipolar axes of cognitive-affect, Table 1. For example, given Ekman's fear category, the proposed scale is: “I feel anxious . . . I feel very confident.” Note that some of these emotions express a similar essence only at opposite ends of the spectrum (such as joy and surprise --the essence is to be low/high in spirits). Since disgust/contempt do not arise frequently in everyday learning settings, we decided not to use these categories. 1.3 Recognizing and responding to student affect through computers The approach of this research is to evaluate learning in classrooms while students work with intelligent tutors and to develop models of student affect along with tools that recognize affect and generate pedagogical interventions. Students are often faced with difficult tasks within computer tutoring situations, tasks which might at times accelerate failure or increase the fear of failure. Recognition of student affect in these situations helps researchers tease apart the learner’s cognitive and affective states and improve tutor intervention. One long-term goal is to help students develop meta-cognitive and meta-affective skills, such as self-awareness and self-regulations for dealing with failure and frustration (Azevedo and Cromley, 2004; Burleson and Picard, 2004; Dweck, 1999). Cognitive Clues Affective Clues Tutor intervention based on inference about a student’s state Student appears curious and focused No intervention needed; Student is engaged in learning and exploration (Flow) Student makes an error Student is frowning, fidgeting, and looking around Alternate actions are needed; Student is confused (Stuck) Evidence of stress, fidgeting, high valence and arousal Alternate actions are needed; Student is under stress (Stuck) Evidence of boredom and confusion Interventions using off-task activities are needed; Student is not engaged (Stuck) Student has not made much progress Student is not frustrated No intervention needed; Student is curious and involved in exploration (Flow) Student is solving problems correctly Student is not frustrated No intervention needed; Student is in control, concentrated and focused (Flow) Table 2: Case studies of students’ cognitive-affective mechanisms Prior research shows that student affect (e.g., frustration or boredom) can be detected within intelligent tutoring systems (McQuiggan and Lester, 2006; Graesser et al., 2007; D’Mello et al., 2007). Our research extends this state-of-the-art by dynamically collecting cognitive and affective information within classrooms, detecting a need for interventions and determining which interventions are most successful for individual students and contexts (problem, affective state). The tutoring system we use responds to students’ cognitive and affective states, see Table 2. If, for example, a student has not exhibited progress in terms of the task, yet sensors indicate that curiosity and exploration (elements of Flow) are at play and related elements of Stuck are not present, the tutor will not intervene; rather will allow the student to further explore the task. One central focus of this research is to generate a framework for long-term pedagogical decisionmaking. Affect recognition can significantly improve a tutor’s long-term planning, e.g., when the tutor allows a student to remain frustrated in the short term. Observing a learner continuously, as a skilled mentor or tutor might do, requires that the computer have affect perception and use that knowledge, along with knowledge about cognitive progress, to reason about a series of student Submitted to special issue on Modeling and Scaffolding Affective Experiences To Impact Learning, International Journal of Learning Technology edited by Rana el Kaliouby and Scotty Craig. 5 actions and interventions, not simply a single-shot action or interaction, but as an ongoing and evolving relationship (Picard et al., 2004; Bickmore and Picard, 2004). In this research, we pay particular attention to understanding learners’ progress from one emotion to another and use dynamic sensor information to interpret objective measures of student progress. Research questions include: • How is affect expressed in student behavior? • How accurate are different machine learning methods (e.g., Bayesian Networks, hidden Markov models) at predicting affect from student behaviors? • How effective are interventions at changing negative affect or changing a state of Stuck into a state of Flow? Can machine learning technology learn reasonable policies for improving student attitude and learning? • How does affect (student emotion and/or computer understanding of it) predict learning? This article discusses a variety of ways that these research questions are addressed, divided into three general areas: affect recognition, interventions that respond to students and development of emotional embodied pedagogical agents. 2 Affect recognition The first area of this research is affect recognition, or use of techniques to detect and evaluate student affect. This research area is fairly new and uses exploratory methods and tools that are likely different from techniques used once the field has matured and reached its steady state. Normal conditions for affect recognition might use non-invasive sensors and machine learning techniques to measure student affect on-line. However, at this early research stage, we use a variety of invasive techniques until we can efficiently predict affect with automatic techniques alone. In one technique described below we invited trained human observers to label students’ affect. Although this technique is labor and time-intensive, it provides several advantages, such as identifying high-level student learning behaviors and suggesting how emotion impacts learning. We induce both static (based on demographics and emotion instruments) and dynamic (based on realtime sensor data as well as inferred hidden variables) student models (McQuiggan & Lester, 2006). Before and after completing the tutoring session (a matter of several days) students are presented with emotion instruments to measure long-term changes in their motivation, self-confidence and boredom. Well-established instruments are employed to measure student emotion before and after interacting with the tutor, Table 3. We use these older instruments because they are verified and used by hundreds of people. These instruments do overlap our cognitive-affective framework; however, we don’t yet know how. Another invasive technique is student self-report, which typically requires interrupting the student during the learning experience and can be unreliable (Picard et al., 2004). Another type of selfreport involves asking students to recall their feelings afterward (Graesser et al., 2007); these techniques are less interruptive but very time consuming and still have high variance in reliability. We explore innovative ways to measure affective states, such as ‘gaming’ or moving rapidly through problems without reading them, or rushing through hints in the hope of being given the answer. It has been estimated that students who game the system learn two thirds of what students who do not game the system learn (Baker et al., 2004). This could be because of frustration, something especially important to detect for students with special needs (Murray et al., 2007). Another possibility is that gaming is a behavior related to poor self-monitoring and/or poor use of meta-cognitive resources. Submitted to special issue on Modeling and Scaffolding Affective Experiences To Impact Learning, International Journal of Learning Technology edited by Rana el Kaliouby and Scotty Craig.